As the Internet has become a commercial service and grown in capacity, the requirements for measuring, monitoring and accounting for IP (Internet Protocol) packet traffic have increased. These requirements can be met using statistics reflecting packets flowing through various networking devices, such as switches, routers, and others. For statistical purposes, network traffic monitored at such devices may be associated with particular users on logical or physical interfaces to the network, and/or specific aggregated traffic trunks. The statistics gathered should reflect both in-bound and out-bound traffic, and be collected at each specific network interface of the networking devices.
Existing systems are available that calculate traffic volume at interfaces, without providing absolute accuracy. When network traffic statistics are used for sensitive applications such as customer billing, however, it is important that every packet is accounted for accurately. Moreover, there are many uses for network traffic statistics that require specific sets of packets to be isolated and counted separately. For example, this is the case in a customer billing application when distinct sets of packets are handled differently to provide different levels of service quality. Similarly, some applications may need to independently and accurately monitor specific packets that violate security policies.
Systems currently exist for detecting detailed information regarding traffic flows within streams of multiple flows at one or more network interfaces. The collected information is used to subsequently create many types of summary records describing traffic statistics. However, existing solutions have significant shortcomings with today's high speed interfaces, due to the fact that even a single traffic stream often includes a large number of flows, each consisting of many packets. As a result, individual devices quickly produce high volumes of statistics to be communicated between system components. Moreover, the raw statistics collected by such existing systems may be sent from multiple devices to a remote system for processing of the aggregated data into one or more application specific formats. Such high volume central aggregation, though potentially flexible, can consume significant communication bandwidth within the network, and requires processing resources in the remote aggregating system, as well as in each statistics gathering device. Alternatively, raw statistics may be processed at a collecting device into a form closer to the needs of a target application, in order to reduce the volume of data transferred to a central processing point. However, requirements that packet devices pre-process large amounts of raw statistics may increase system processing requirements which may greatly increase the cost and system power requirements.
Lack of accuracy is another problem to be avoided in statistics gathering and processing. Packet forwarding devices having high speed interfaces use fast, powerful processors and/or dedicated hardware to forward common types of packets. Packet types requiring more complex processing may be passed to a separate subsystem running an exceptions handling process. In existing systems, packets destined for such exception processing subsystems may bypass the statistics collecting mechanism. Moreover, accurate statistics collection may require creation of specific statistics data that can be used to adjust general statistics records to improve overall accuracy. For example, in some cases, it may be desirable to subdivide flows for statistical purposes based on conformance to a traffic policing policy or rate contract. To maintain useful statistics, information regarding conformance of specific packets relative to other packets within a shared flow needs to be accessible. This information could then be relayed to the statistics processing system.
One possible approach to improving efficiency is to employ data reduction algorithms that rely on historically observed statistical traffic patterns. However, since traffic patterns often change over time, these types of statistical techniques may give rise to non-deterministic behaviors.
Accordingly, it would be desirable to have a new system for generating meaningful statistics on network traffic that is fully deterministic, and that avoids inaccuracy and poor scalability problems. The system should be convenient for use with common Internet protocols, including IP, MPLS (MultiProtocol Label Switching), and Ethernet. The system should not adversely impact forwarding operation performance in traffic monitoring systems, and should accurately gather network traffic information reflecting all packets regardless of type, without ignoring packets in the face of high traffic levels.